Five questions, ~3 minutes

1. Who calls this — internal services or end users?
   (decides auth model, SLA, abuse surface)
2. Scale? Peak QPS, total entities, growth rate.
   (decides sharding and whether one box works)
3. Latency SLO at p99, and is it on the critical path?
   (decides sync vs async, cache vs no cache)
4. Consistency: can a reader see stale data, and for how long?
   (the single biggest architecture fork)
5. One region or global? Where are the writers?
   (decides replication topology and conflict handling)

Then: "I'll assume X, Y, Z unless you want to steer me." Stating assumptions out loud is what converts silence into a grade.

Then the shape of the hour

• ~10 min clarify + capacity math out loud.
• ~10 min high-level boxes only. Resist diving.
• ~25 min deep dive — the interviewer picks the box. Have the atomic sequence ready for the one that has a race in it.
• ~10 min failure & trade-off. Volunteer this; don't wait to be asked.
• ~5 min evolution: MVP → 10×, and where you'd re-architect.

Verbal-round adaptation: when there's no whiteboard, narrate structure explicitly — "three tiers: edge, decision, storage; let me walk the write path first, then the read path." Without a diagram the interviewer can only grade the words, so the words have to carry the structure.

Say these unprompted

• Idempotency key on every retryable write. Names the double-write race before it's asked.
• Fail-open vs fail-closed, tied to the business consequence, not to taste.
• Hot key mitigation: shard the key (key:{0..N}), or local pre-aggregate then flush.
• Backpressure: what the caller sees when the queue is full. 429 with Retry-After beats silent drop.
• Watermarks / bounded state for anything streaming, so state doesn't grow forever.

Capacity math you should have memorized

Thing	Number
Redis point op, same DC	~0.5–1 ms p99, ~100k ops/s/core
SSD random read	~100 µs; ~50–100k IOPS/device
Cross-region RTT (US↔EU)	~80–100 ms
Spanner cross-region commit	~50–150 ms
Kafka partition	~10 MB/s sustained
1 day at 100k QPS	~8.6 B events
Object storage	~$20/TB/mo hot, ~$4/TB/mo cold

Functional

• Producer services submit a notification event; the platform decides channel, renders, delivers, retries.
• Channels: mobile push (APNs/FCM), email, SMS, in-app inbox. Fallback chain per event type.
• User preferences and quiet hours honored; unsubscribe is legally binding for email/SMS.
• Fan-out: one event may target one user, a segment, or all users (announcement).
• Exactly-once user-visible delivery: at-least-once transport + dedup at the edge.

Out of scope: content authoring UI, ML send-time optimization, marketing campaign scheduling.

Non-functional

• 10⁸ users; steady ~50k notifications/s, burst 1 M/s on a broadcast.
• Transactional (2FA, security alert): p99 < 5 s end-to-end. Marketing: minutes are fine.
• Durability > latency for transactional — never silently drop.
• Multi-region, survives loss of one region and of any single third-party vendor.

Core tension: the burst is 20× steady and the third-party channels (APNs, an SMS vendor) are both rate-limited and unreliable. So the system's real job is absorbing bursts and metabolizing partner failure, not sending messages.

Producer (write path)

POST /v1/notify
Idempotency-Key: {event_id}
{ type, audience: {user_id | segment_id},
  payload: {...}, priority, ttl_sec }
→ 202 { event_id, status: "ACCEPTED" }

POST /v1/notify:broadcast
{ type, segment_id, payload, rate_limit_per_sec }
→ 202 { event_id, estimated_recipients }

202, never 200. Accepting into a durable log and returning immediately is what decouples the producer from APNs being down. A producer that blocks on delivery has coupled its own availability to a third party.

Consumer & ops (read path)

GET  /v1/inbox?user_id=&cursor=   // in-app feed
POST /v1/deliveries/{id}:ack     // client read receipt
GET  /v1/events/{event_id}       // fan-out status rollup
     → { total, sent, delivered, failed, suppressed }

POST /v1/webhooks/vendor/{vendor}  // bounce/delivery callbacks
PUT  /v1/prefs/{user_id}
POST /v1/unsubscribe   { token }   // one-click, no login

Two fan-out modes

Targeted (99% of events, 1..1000 users):
  fan-out inline in the worker
  write Delivery rows in a batch
  enqueue per channel

Broadcast (segment or ALL, 10^8 users):
  DO NOT expand in one worker
  1. materialize segment -> sharded user-id ranges
  2. emit N "fan-out chunk" tasks (chunk = 10k users)
  3. each chunk expands independently, resumable
     via (event_id, chunk_id) checkpoint
  4. token-bucket the emit rate per channel

A broadcast at 10⁸ users writing 10⁸ Delivery rows is ~100 GB and hours of vendor throughput. It must be a paced background job, not a spike.

Why chunking, and where it still hurts

Chunk-level checkpoints make fan-out resumable and idempotent: a worker crash replays one 10k chunk, and the deterministic delivery_id means the replayed rows collide with the originals instead of duplicating. Without this, a crash 80% through a 10⁸ fan-out leaves you with no safe action.

Defend it — the priority inversion. A broadcast dumped into the same queue as 2FA codes will bury them behind hours of backlog, and a 2FA code delivered in 40 minutes is worse than useless. Fix: separate physical queues per priority class, not just a priority field. Transactional gets its own partitions, its own worker pool, and a reserved share of vendor rate budget that bulk traffic can never borrow. This is the single most likely follow-up in this round; volunteer it.

The chain of idempotency

1. Producer supplies event_id (Idempotency-Key).
   Ingest does INSERT ... IF NOT EXISTS.
   Retry of the same HTTP call -> same event, no dup.

2. delivery_id = hash(event_id, user_id, channel)
   Fan-out replay writes the same row. Dedup by PK.

3. Send is guarded by a conditional state transition:
     UPDATE Delivery
        SET state='SENDING', attempt=attempt+1
      WHERE delivery_id=? AND state IN ('PENDING','FAILED')
   Only the winner calls the vendor.

4. Vendor call carries its own idempotency token
   where supported (SES, Twilio); where not, accept
   at-least-once and dedup on the DEVICE by event_id.

Why true exactly-once is impossible, and the honest answer

Step 3 has an unavoidable gap: you set SENDING, call the vendor, and crash before recording the result. You cannot know whether the vendor sent it. Any system that claims exactly-once across a third-party boundary is lying.

So decide per channel, by consequence. Push and in-app: retry (duplicate push is mildly annoying, and the client dedups on event_id anyway). Email: retry (dup email is tolerable). SMS: do not blind-retry — it costs real money and duplicate 2FA codes confuse users and can invalidate the first code. For SMS, reconcile against the vendor's delivery-status API before a retry, and cap at one reconciliation attempt.

Belt-and-suspenders: the client-side dedup on event_id with a 24 h local cache is what actually delivers the user-visible guarantee. Server-side you only ever promise at-least-once.

Retry policy that doesn't amplify an outage

backoff = min(2^attempt * 1s, 15m) * jitter(0.5..1.5)
max_attempts = 5, then DLQ
hard stop at event.ttl_sec  -- a 2FA code has ttl 300s;
                            -- retrying at t+10m is harmful

Circuit breaker per (channel, vendor):
  error_rate > 50% over 30s  -> OPEN
  OPEN: stop calling, park in queue, alarm
  half-open probe every 10s

Jitter is not optional. Without it, a vendor recovering from a 60 s outage gets every parked message at the same instant and immediately falls over again.

Channel fallback — and why it's usually wrong

The prompt asks for multi-channel fallback, so propose it, then bound it. Fallback chain lives on the event type: SECURITY_ALERT: [push, sms, email], PROMO: [push] (never escalate marketing to SMS — it costs money and generates complaints).

Defend it: escalating on "not delivered within N seconds" is dangerous, because push delivery receipts are unreliable and slow — you'll double-notify constantly. Only escalate on a hard signal: no registered device token, token rejected by APNs, or circuit breaker open for that channel. Timeout-based escalation should have a generous floor (minutes, not seconds) and must be off for anything bulk.

Vendor redundancy: two vendors per paid channel, weighted routing, automatic weight shift when a breaker opens. This is the concrete answer to "SMS vendor dies" and it's cheap to say.

Throughput

• Steady 50k notifications/s → ~4 B/day. At ~1 KB of Delivery row each, ~4 TB/day before TTL.
• 30-day TTL on Delivery → ~120 TB. Wide-column with compression, ~20 nodes. Cheap.
• Kafka: 50k msg/s × 1 KB = 50 MB/s → ~10 partitions at 10 MB/s, call it 32 for headroom and priority isolation.
• Broadcast to 10⁸: at a paced 100k sends/s that's ~17 minutes per channel — quote this number, it reframes "real-time" honestly.

Latency budget (transactional, 5 s p99)

• Ingest + commit to log: ~20 ms
• Log → fan-out worker pickup: ~100 ms
• Prefs lookup (cached): ~2 ms
• Delivery write + queue: ~30 ms
• Vendor call (APNs): ~200 ms p99
• Device receipt: ~1 s, outside our control
• Slack: ~3.6 s — which is what pays for exactly one retry inside the SLA.

Functional

• Ingest multi-dimensional time series: metric{service,region,host,env,segment} → value@ts.
• Query: aggregate over label selectors and time ranges, for dashboards and detectors.
• Detection, at least two mechanisms: static threshold and dynamic baseline (seasonality / change-point).
• Alerting: routing by team, dedup, grouping, silencing, escalation, multiple channels.
• Visualization: dashboards, plus the ability to slice by dimension after an alert fires.

Out of scope: log search, distributed tracing, incident management workflow (page a PagerDuty-equivalent, don't build it).

Non-functional

• 10⁷ active series, 10⁷ datapoints/s ingest.
• Detection latency < 60 s from event to page for critical metrics.
• Query p99 < 1 s for a 24 h dashboard panel.
• The monitoring system must not depend on the systems it monitors, and must degrade before it lies.

Core tension: alert quality is a precision/recall trade-off, and it is a product decision, not a modelling one. A detector that never misses will page so often that humans stop reading, at which point recall is effectively zero. Everything below is about being deliberately less sensitive in exchange for being believed.

Write path

POST /v1/ingest            (protobuf, batched, gzip)
{ samples: [{ labels{}, ts, value }, ... ] }
→ 204, or 429 with Retry-After when over quota

// agents push every 10s; server does NOT pull.
// per-tenant series-cardinality quota enforced HERE.

Push, not pull, at this scale — pull requires the monitoring system to hold a service-discovery view of 10⁷ targets and creates a fan-in storm. Pull's advantage (you know when a target is missing) is recovered by a staleness detector.

Read / control path

GET /v1/query_range
  ?selector=http_errors{service="ads",env="prod"}
  &start=&end=&step=60s&agg=sum by (region)
→ { series: [{labels, points:[[ts,v]...]}] }

PUT  /v1/detectors/{id}
POST /v1/silences  { matcher, until, reason }
GET  /v1/alerts?state=FIRING
POST /v1/detectors/{id}:backtest  { window: "30d" }

Backtest is the feature that makes detectors shippable — replay a candidate detector over 30 days of history and show how many times it would have paged. Nobody should tune a threshold in production.

Static threshold (streaming)

for each eval tick (15s):
  v = agg(selector, last 5m)
  if compare(v, threshold):
     pending[fp] = pending[fp] or now
     if now - pending[fp] >= for_duration:   # e.g. 5m
        emit FIRING
  else:
     clear pending[fp]; emit RESOLVED if firing

for_duration is the anti-flap knob, and it's the whole reason threshold alerting is usable. A metric crossing a line for 15 s is noise; crossing it for 5 minutes is an incident. Cost: you've added for_duration to your detection latency, so critical detectors run a short duration on a tight threshold and a long duration on a loose one.

Dynamic baseline (batch)

Seasonal decomposition per series:
  baseline(t) = median over last K weeks
                at same (weekday, time-of-day)
  band(t)     = baseline(t) ± k * MAD(t)
  fire when actual outside band for for_duration

Change point (fast, cheap):
  compare rolling mean/var of last 5m vs prior 60m
  robust z = (m1 - m0) / (1.4826 * MAD_0)
  |z| > 4  -> candidate

Median + MAD, not mean + stddev. Mean and standard deviation are wrecked by the very outliers you're trying to detect — one past incident inflates the band and hides the next one. MAD is robust to ~50% contamination, which is the difference between a detector that degrades gracefully and one that silently stops working after its first real incident.

How these systems really die

Not from datapoint volume — from series count. Every distinct label combination is a new series with its own index entry and memory-resident head block. One engineer adds user_id or request_id or a raw URL as a label and cardinality goes from 10⁷ to 10⁹ overnight, the index blows out of RAM, and the whole cluster falls over. This takes down monitoring for everyone, during whatever incident prompted them to add the label.

Defenses, in order:
1. Per-tenant series quota, enforced at INGEST.
   Over quota -> reject new series, keep existing.
   (Never reject existing: partial data is worse
    than no new dimensions.)
2. Label-value cardinality cap per label name.
   >1000 distinct values -> auto-drop label, warn.
3. Cardinality explosion detector on the metadata
   itself: d(series_count)/dt alert.
4. Chargeback: teams see their series cost.

The other structural risks

Hot shard: hashing by series_id spreads writes evenly by construction, but a query for sum by (region) over a million series fans out to every shard. Mitigate with pre-computed recording rules — materialize the common aggregations at write time so dashboards read one cheap series instead of a million expensive ones.

Alert storm: one bad deploy fires 400 detectors at once and pages a human 400 times. The alert manager must group by a shared label set (e.g. all alerts for service=ads,region=us-east become one notification with 400 lines) and apply an inhibition rule — if ServiceDown is firing, suppress every dependent HighLatency alert for that service. Without inhibition rules the system is technically correct and operationally useless.

Silent failure: the worst outcome isn't a false page, it's no page because ingest stopped. Hence the dead-man's switch below.

Staleness > absence

An absent series and a healthy series look identical to a threshold detector: neither crosses the line. So every detector needs an explicit staleness clause — absent_over_time(selector[5m]) fires its own alert. Otherwise a crashed agent reads as "all clear," which is the single most dangerous bug a monitoring system can have.

Dead-man's switch: a synthetic detector that fires continuously and routes to an external service which pages you when the heartbeat stops. It's the only construct that catches "the monitoring system itself is down," and it costs about ten lines.

Circular dependency

If the monitoring stack runs on the same Kubernetes control plane, the same service mesh, and the same object store as production, then a production outage blinds you exactly when you need sight. State this explicitly: the meta-monitoring stack runs in a separate failure domain — different region, different account, minimal dependencies, and its notification path does not traverse the primary network. It monitors ~50 series about the main system rather than 10⁷, so it can be small and boring.

Degrade before lying: under ingest overload, shed low-priority tenants and keep critical ones at full fidelity, rather than uniformly downsampling everyone. Uniform degradation quietly reduces detection sensitivity across the board with no signal that it happened.

Ingest & storage

• 10⁷ series × 1 sample / 10 s = 10⁶ datapoints/s steady; size for 10⁷ peak.
• At ~1.4 bytes/sample compressed: 10⁶/s → ~120 GB/day raw-resolution.
• 15-day raw retention → ~1.8 TB. Genuinely small — the compression is the headline number.
• Index: ~10⁷ series × ~200 B = 2 GB of index in RAM, which is why cardinality (not volume) is the binding constraint.
• Ingest node handles ~200k samples/s → ~50 nodes at peak.

Detection latency budget (60 s)

• Agent scrape interval: 10 s (worst-case age at emit)
• Agent → ingest → log commit: ~2 s
• Streaming detector eval tick: 15 s
• Alert manager group_wait: 10 s (batches the storm)
• Notifier → page: ~3 s
• ~40 s, leaving ~20 s of slack — which is exactly the budget that for_duration eats, so critical detectors run for_duration=0 on high thresholds.

Functional

• Owner sees near-real-time location of their devices from web or another device.
• Lost mode: play sound, mark as lost with a message, remote lock, remote wipe.
• Offline devices: report position when they come back online; optionally crowd-sourced reporting via nearby devices.
• Family sharing: explicitly authorized members can view, with revocation.
• Audit log the owner can read: who queried my location, when.

Out of scope: the map rendering stack, device activation lock enforcement, carrier integration.

Non-functional

• 10⁸ devices; peak 10⁶ location updates/sec.
• Query QPS is tiny by comparison — people check rarely. Write-dominated by ~1000:1.
• Battery: location reporting must cost single-digit mW average.
• The server must not be able to read device locations. Treat that as a hard requirement, not a nice-to-have.

Core tension: a normal design puts plaintext locations in a database so the service can index and query them efficiently. But a database of where 10⁸ people are is the highest-value breach target you could build, and an insider-abuse vector. The entire design is about giving that up — and then paying for it in query complexity and lost server-side features.

Report path (huge volume, unauthenticated by design)

POST /v1/reports        // from ANY device, incl. finders
{ report_key, ciphertext, ts }
→ 204

// No auth header tied to identity — attaching one
// would let the server correlate reporter to subject.
// Abuse control is anonymous rate-limiting:
// device attestation token + per-token quota.

POST /v1/commands/{device_id}
Signed-By: owner_key
{ type: LOCK, nonce, expires_at, signature }
→ 202 { command_id }

Query path (tiny volume)

POST /v1/reports:fetch
{ report_keys: [k1..kN] }     // client-derived
→ { found: [{key, ciphertext, received_at}] }
// client decrypts locally; server sees nothing

GET  /v1/devices               // my devices + status
POST /v1/grants  { grantee, device_id, expires_at }
DELETE /v1/grants/{id}
GET  /v1/audit?device_id=      // who looked, when

Fetch is a multi-get of opaque keys. The server can't tell whose device it is, or even that the N keys belong to one device. That property is the whole design.

Rotating keys, derived by both sides

At pairing (owner device + lost device share a secret):
  master_secret  ms          (never leaves either)

Every 15 minutes, epoch i:
  sk_i  = KDF(ms, i)                  # both sides derive
  pk_i  = curve_pub(sk_i)             # advertised over BLE
  key_i = SHA256(pk_i)                # the report_key

Finder:
  sees pk_i over BLE
  loc_ct = ECIES_encrypt(pk_i, {lat,lng,acc,ts})
  POST { report_key: SHA256(pk_i), ciphertext: loc_ct }

Owner looking for the device:
  for i in last 7 days (672 epochs @15m):
      keys.append(SHA256(pk_i))
  POST /reports:fetch { keys }        # ~672 keys
  decrypt each hit with sk_i          # locally

What each property buys, and what it costs

• Server can't decrypt — it never has any private key. A full database dump leaks nothing but "some blobs existed."
• Server can't link reports to a device — pk_i rotates every 15 min, so consecutive reports look unrelated. This is what stops the server (or an insider) reconstructing anyone's movement history.
• Finder can't track the lost device — it sees only a rotating pseudonym and can't decrypt what it just uploaded.
• Stalking defense — receiver devices detect an unknown beacon that persists across epochs while moving with you and alert the user. This is a real, shipped requirement and volunteering it is a strong signal.

Cost, stated honestly: ~672 key lookups per query instead of one; no server-side features at all (no "notify me when it moves", no geofencing, no server-side history search); and if the owner loses every device holding the master secret, past reports are unrecoverable. That last one is a genuine product trade and worth naming.

Sequence

1. Owner authenticates (with 2FA) to the account service.
2. Client builds:
     cmd = {device_id, WIPE, nonce, expires_at}
     sig = sign(owner_private_key, cmd)
3. Server stores cmd, does NOT need to trust itself:
     it also checks the owner's session, but that is
     defense in depth, not the control.
4. Push wakes the device.
5. DEVICE verifies sig against the owner pubkey it
   pinned at enrollment, checks nonce not seen and
   expires_at in the future, then executes.
6. Device ACKs with a signed receipt -> audit log.

Why verification must happen on the device

If the server decides who may wipe, then a server compromise, a malicious insider, or a support-tool bug can wipe 10⁸ devices. Verifying the owner's signature on the device reduces the server to an untrusted transport: the worst it can do is refuse to deliver commands (a denial of service, recoverable) rather than forge them (catastrophic, irreversible).

Defend it — replay. Without nonce + expires_at, anyone who captures a valid LOCK command can replay it forever. The device keeps a small seen-nonce window and rejects anything expired, which bounds that memory.

Defend it — the offline wipe race. A stolen device that's kept offline never gets the command. WIPE is queued with a long TTL and fires the moment it gets network; meanwhile the real protection is local full-disk encryption, which the remote wipe merely accelerates by destroying the key. Say this — candidates who present remote wipe as the primary protection are wrong, and the interviewer knows it.

Battery on the device

Choice	Why
BLE advertise, don't GPS-fix	The lost device only broadcasts a rotating pubkey — no GPS, no radio uplink. Micro-amp scale; a dead-battery phone can beacon for hours on reserve power.
Finder supplies the location	The finder was already computing its own position. Marginal cost ≈ one small HTTPS POST, batched.
Batch + opportunistic upload	Finders buffer reports and flush on Wi-Fi / when the radio is already up. Never wake the modem just for this.
Adaptive rate on the owner's own devices	Report position on significant-change, not on a timer. Stationary device → near-zero traffic.

Efficiency on the server

Write path is 10⁶/s of ~200-byte blind appends with no index maintenance and no read-modify-write — the cheapest possible write. Sharded by report_key, which is a hash, so distribution is uniform by construction and there is no hot key possible: a popular location doesn't concentrate, because keys are per-device-epoch, not per-place.

Dedup: twenty phones in a train carriage all report the same beacon in the same epoch. Twenty near-identical ciphertexts under one key. Cap at ~K reports per (key, epoch) — first-K-wins — and drop the rest at ingest. That's a ~10× write reduction in dense areas and it costs nothing, since the owner only needs one good fix per epoch.

7-day TTL is doing heavy lifting: it bounds storage, and it bounds the blast radius of any future cryptographic weakness. Longer retention has no product value — nobody finds a phone with a 3-week-old fix.

Write volume

• 10⁶ reports/s × ~250 B = 250 MB/s ingest, ~21 TB/day.
• With per-epoch dedup at K=5 in dense areas: ~10× reduction → ~2 TB/day effective.
• 7-day TTL → ~15 TB steady state. Small enough to keep entirely on SSD.
• Shard at ~50k writes/s/node → ~20 ingest shards at peak. Uniform by construction.

Read & crypto

• Queries: say 10⁸ users × 1 check/week ≈ 165 QPS. Three orders of magnitude below writes — which justifies optimizing everything for write.
• 672 keys/fetch × 165 QPS = ~110k key lookups/s. Trivial for a hash-sharded KV.
• ECIES encrypt on a finder: ~1 ms, negligible against a BLE scan already running.
• Key derivation on the owner client: 672 × KDF ≈ tens of ms, done once per open.

Functional

• Fleet vehicles capture panoramic frames + GPS/IMU metadata and upload them.
• Uploads survive intermittent connectivity: resumable, deduplicated, eventually complete.
• Ingested images are durably stored and published to downstream consumers (blur/PII, image understanding, tiling, map generation).
• Operators can see per-vehicle ingest status and re-drive coverage gaps.

Out of scope: the ML models themselves, the map-tile serving CDN, camera firmware.

Non-functional

• 10k vehicles; each ~1 frame/sec while driving, ~8 h/day. ~10 MB/panorama.
• Ingest is throughput-bound, not latency-bound — nobody needs the photo in a second.
• Durability is the hard requirement: a lost frame means re-driving a street. Target 11 nines.
• Cost matters at petabyte scale; storage tiering is part of the design, not an afterthought.

Core tension: the client is a moving vehicle on flaky cellular, generating far more data than its link can carry, and it cannot be trusted (a stolen credential lets someone poison the map). So the design is really about a well-behaved offline-first client plus an untrusted-upload security model, with the storage tier being the easy part.

Upload path (resumable, the actual question)

POST /v1/uploads:init
Authorization: Bearer {short-lived vehicle token}
{ session_id, seq, content_hash, bytes, captured_at,
  lat, lng, heading }
→ 200 { upload_id,
        upload_url,        // signed, direct-to-storage
        expires_in: 3600,
        already_have: false }   // <-- dedup short-circuit

PUT {upload_url}
Content-Range: bytes 5242880-10485759/10485760
→ 308 { received: "0-10485759" }   // resume point
→ 201 when complete

GET /v1/uploads/{upload_id}   // "where was I?"
→ { received_ranges: [[0, 5242879]] }

Downstream & ops

// consumers subscribe, they don't poll
topic: frames.uploaded  { frame_id, blob_uri, geo }
topic: frames.published { frame_id, after blur+verify }

GET /v1/coverage?s2_cell=&since=
→ { frames, gaps: [{lat,lng,reason}] }
GET /v1/sessions/{id}/status
→ { captured: 28800, uploaded: 28112, pending: 688 }
POST /v1/frames/{id}:quarantine  { reason }

Answering the "upload response design" follow-up: return 308 with the exact committed byte range, plus already_have on init so a client that crashed after uploading never re-sends 10 MB. The response's job is to make the client's retry decision unambiguous.

The client is the interesting system

Capture loop (never blocks on network):
  frame -> local SSD spool + WAL entry
  compute sha256 while writing
  spool is a bounded ring: 500 GB ≈ 14 h of capture

Upload loop (independent, opportunistic):
  while spool not empty:
    pick oldest UNSENT frame
    if link_quality == CELLULAR and not urgent:
        upload at throttled rate (leave headroom)
    if link == WIFI (depot / known AP):
        upload at full rate, drain aggressively
    on 5xx / timeout:
        exponential backoff + jitter, keep frame
    on 201:
        mark SENT, delete local copy lazily
        (keep until VERIFIED event or 24h)

Chunk size adapts: start 8 MB, halve on each
timeout down to 256 KB, grow back on success.

Why this shape

Capture and upload must be decoupled queues. If capture blocks on upload, a tunnel or a dead cell means lost coverage and a re-drive — the single most expensive failure in this system. Spooling to local disk turns a network problem into a storage problem, which is far cheaper.

Adaptive chunk size is the concrete answer to "network is bad." On a marginal link, a 10 MB PUT that dies at 90% wastes 9 MB of the vehicle's uplink; 256 KB chunks lose almost nothing per failure. The trade is more round-trips and more per-request overhead, so you grow the chunk back when the link proves itself.

Deletion is delayed until VERIFIED, not until the 201. A 201 means bytes landed; it doesn't mean the server rehashed them successfully. Deleting the only copy on a 201 is how you lose data to a silent corruption. Cost: local disk holds a day of already-uploaded frames.

Belt-and-suspenders — the bounded spool. 14 hours of buffer exceeds an 8-hour shift, so a vehicle that never sees network all day still loses nothing; it drains at the depot on Wi-Fi. If the ring does wrap, drop by lowest marginal coverage value (a frame from an already-well-covered cell) rather than oldest-first. Volunteering a smarter eviction policy than FIFO reads well.

Layered credentials

Layer 1 — hardware identity (long-lived, never sent)
  Per-vehicle key in a TPM/secure element at fleet
  provisioning. Private key cannot be extracted.

Layer 2 — short-lived access token (the thing on the wire)
  Vehicle signs a challenge with the TPM key
    -> token service returns JWT, TTL 1 hour,
       scoped: {vehicle_id, session_id, WRITE only,
                geo_bbox of today's route}
  Never a long-lived API key on the device.

Layer 3 — per-object signed URL (TTL 1 hour, single path)
  Scoped to exactly /raw/{cell}/{date}/{content_hash}
  So even a leaked URL can write ONE object,
  whose path is the hash of its own contents —
  so it cannot write anything but that exact image.

"The token is compromised. What do you do?"

Answer in the order the interviewer is grading: contain, revoke, detect, prevent.

• Blast radius is already small by design — the token is write-only, expires in an hour, is scoped to one vehicle and one geographic box, and the signed URLs it can obtain are content-addressed. An attacker cannot read anyone's imagery, cannot overwrite existing frames with different bytes, and cannot write outside today's route area.
• Revoke: add the token's jti to a deny list checked at the ingest control plane (small, because TTL is short — the list self-cleans in an hour). Revoke the vehicle's provisioning cert to stop it minting new tokens.
• Detect: the signals that catch this are behavioral — uploads from an IP/ASN that doesn't match the vehicle's cellular carrier, frames whose GPS is inconsistent with the session's route continuity or physically impossible speed, upload rate exceeding the camera's capture rate, or two concurrent sessions for one vehicle_id. Any of these quarantines the frames rather than dropping them, so a false positive is recoverable.
• Prevent: mutual TLS with the device cert on the control plane, so the bearer token alone is insufficient; anything anomalous goes to QUARANTINED and is human-reviewed before publish.

The point to land: the right answer to a compromised credential isn't a faster revocation loop, it's designing so that a compromised credential can't do much. Short TTL, narrow scope, content-addressing, and write-only are all doing that work.

Event-driven, not synchronous

Ingest publishes frames.uploaded and stops caring. Consumers — blur, image understanding, tiling, map generation — subscribe independently with their own offsets, their own scaling, and their own failure domains. A stalled ML pipeline creates consumer lag, not upload failures.

Reprocessing is the reason raw is immutable and kept forever. Models improve; when a better sign-detection model ships you replay the last N years of frames.uploaded from the log (or scan the metadata store by S2 cell) rather than re-driving the planet. Derived artifacts — tiles, extracted geometry — are therefore disposable, and treating them that way is what makes the storage bill defensible.

The blur stage must be a gate, not a peer

Faces and license plates must be blurred before anything user-facing exists. If blur is just another parallel subscriber, there's a window where the tiling pipeline has already published an unblurred panorama — a privacy incident with legal consequences in the EU and elsewhere.

So: two topics. frames.uploaded is consumed only by blur/verify. Blur writes a redacted derivative and emits frames.published; everything user-facing subscribes to that one. Internal-only consumers may read raw under separate authorization and audit.

Defend it: blur models miss things, so you need a takedown path — a user reports their face, you re-blur that frame and purge derived tiles, which is only tractable because frame_id → tile lineage is recorded. Without lineage tracking, a single takedown request means rebuilding a region. Say this; it's the kind of operational detail that separates L5 from L4 here.

Ingest volume

• 10k vehicles × 1 frame/s × 8 h = 288 M frames/day.
• At 10 MB each: ~2.9 PB/day raw. This number is the design constraint — say it early.
• Sustained ingest if spread over 24 h: ~33 GB/s. Per vehicle that's only ~10 Mbps, which is exactly why depot Wi-Fi drain matters — cellular can't carry it all.
• Control plane: ~3.3k init QPS average, ~10k peak. Trivially small next to the byte volume.

Storage & metadata

• ~1 EB/year raw. With lifecycle tiering (30d hot / 1y nearline / then coldline) the blended rate is roughly 4–5× cheaper than all-hot.
• Metadata: 288 M rows/day × ~500 B = ~145 GB/day — 0.005% of the blob volume, which is why it can afford indexes.
• Vehicle spool: 500 GB SSD ÷ 10 MB = 50k frames ≈ 14 h of capture, comfortably more than one shift.
• Depot drain: 14 h of backlog at 1 Gbps Wi-Fi ≈ 70 min. Sizes the depot link.

Functional

• HQ authors a global menu; countries/regions/stores override parts of it (price, availability, language, local items).
• Menus vary by daypart: breakfast / lunch / dinner, and can differ day to day.
• Push updates to heterogeneous devices: digital menu boards, kiosks, tablets, drive-thru displays, the mobile app, and third-party delivery aggregators.
• Scheduled activation ("this menu goes live Monday 6am local") and emergency rollback.
• A store must show a correct menu even with no network.

Out of scope: order placement, inventory/POS integration beyond an availability flag, the authoring UI itself.

Non-functional

• 40k stores × ~5 devices = 200k endpoints, ~100 countries.
• Update propagation: minutes is fine for planned changes; seconds matter for a takedown (allergen recall, wrong price).
• Availability >> consistency at the edge: a device must never show a blank screen.
• Correctness of price is the one thing that's legally and financially sensitive.

Core tension: the update volume is laughably small — this is not a scale problem. It's a configuration-correctness and safe-rollout problem across 200k unreliable, heterogeneous, intermittently-connected endpoints. Candidates who reach for Kafka and sharding here have misread the question.

Device path (pull, not push)

GET /v1/devices/{device_id}/manifest
If-None-Match: {current_version_id}
→ 304 Not Modified                      // the 99.9% case
→ 200 {
    schedule: [
      { daypart: "breakfast", from: "06:00",
        to: "10:30", version_id: "a1b2",
        bundle_url: "https://cdn/.../a1b2.tar",
        sha256: "...", size: 4_200_000 },
      { daypart: "lunch", ... }
    ],
    poll_after_sec: 300,
    emergency_channel: "wss://..."
  }

POST /v1/devices/{device_id}/heartbeat
{ active_version_id, staged_version_ids[],
  render_errors[], clock_skew_ms }

Devices pull on a jittered interval and hold a long-lived websocket only for emergencies. Pull scales trivially through a CDN and survives a device being offline for a week; push-only would require the server to track 200k connections just to deliver a change that isn't urgent.

Authoring & ops path

PUT  /v1/layers/{layer_id}      { patch }
POST /v1/versions:preview
  { layer_set, daypart, store_id }
  → fully resolved menu + diff vs current

POST /v1/versions/{id}:approve   // 2-person for price
POST /v1/rollouts
  { version_id, cohort: "canary_50_stores",
    effective_from: "2026-03-01T06:00 LOCAL" }
POST /v1/rollouts/{id}:halt
POST /v1/emergency/takedown
  { item_id, scope, reason }     // seconds, not minutes

GET  /v1/fleet/status?version_id=
  → { on_target: 39_812, stale: 188, unreachable: 40 }

Never activate over the network

Publish (T-24h or more):
  resolver produces bundles for every
  (store, daypart) combination for the next 48h
  → manifest lists them ALL with local times

Store gateway (continuously):
  downloads every staged bundle
  verifies sha256
  keeps last-known-good + all staged

Activation (exactly at 06:00 store-local):
  gateway compares local clock to schedule
  swaps active pointer atomically
  devices re-render from local files

Network down at 06:00?  Still switches.
Network down for 3 days? Runs out of staged
  versions → holds last-known-good, alarms.

Why this is the crux of the question

The prompt's "早中晚餐可能跟前一天不一样" is a trap for anyone who designs a push-at-activation-time system. If the breakfast menu is pushed at 06:00 local, then every store in a timezone hits you simultaneously — a thundering herd at every hour boundary around the globe — and any store whose link is down at that moment shows the wrong menu during its busiest period. Pre-staging converts a synchronized network event into an unsynchronized download plus a local clock comparison.

Defend it — clock skew. The design now depends on device clocks. A gateway with a drifting clock switches to dinner at lunchtime. Mitigations: NTP on the gateway, clock skew reported in every heartbeat with an alert above ~60 s, and a sanity rule that the gateway won't jump more than one daypart forward without a server confirmation. This is the failure mode the interviewer will hunt for once you propose local activation, so raise it yourself.

Timezones and DST: schedules are stored as local wall-clock times with an IANA timezone ID per store, never as UTC offsets — offsets break twice a year, and a store in a DST-shifting region would show breakfast an hour late every spring.

Ship data, plus per-capability renditions

Bundle contents:
  menu.json          -- semantic content, device-agnostic
                        items, prices, i18n strings, order
  layouts/
    board_16x9.json     kiosk_portrait.json
    drive_thru.json     app_compact.json
  assets/
    {hash}.webp @1x/2x   {hash}.mp4 (boards only)
  render_hints: { max_items_per_panel, font_scale }

Device fetches ONLY the layout + assets matching
its registered capabilities → a drive-thru display
never downloads 4K video it can't show.

The trade

The alternative — pre-rendering images per device type server-side — makes devices trivially dumb but multiplies bundle count by device type, and any layout tweak means republishing everything. Shipping semantic JSON plus layout descriptors means one content pipeline and per-device presentation, at the cost of some rendering logic on the endpoint.

Defend it — the capability lie. Devices misreport capabilities, and old firmware won't understand a new layout field. So: version the bundle schema, require devices to declare a supported schema range in the manifest request, and have the server serve the highest bundle version that device can parse. Unknown fields must be ignored, not fatal. Without this, one new field bricks every unpatched screen in the fleet — and menu boards get firmware updates roughly never.

Third-party aggregators (delivery apps) are just another consumer type, but they pull via API rather than bundle, and they're the one consumer where a stale price becomes a customer-facing refund. Give them a webhook on version change plus a short cache TTL.

Two speeds, deliberately

	Planned change	Emergency takedown
Trigger	Approved version, scheduled	Allergen error, wrong price, recall
Path	CDN pull, jittered, staged	Websocket push + poll interval collapses to 10 s
Granularity	Whole version	Single item_id: hide it
Target	Minutes–hours	< 30 s to 99% of fleet
Fallback	—	If push fails, next poll catches it

A takedown is a subtractive overlay, not a new version — "hide item X everywhere" is a tiny message that any device can apply to whatever bundle it's currently running, including a stale one. That's why it's fast and why it works on devices that haven't synced.

Canary the way you'd canary code

A bad menu version is a production incident with revenue impact. Roll out by cohort: 50 stores → one region → country → global, with automatic halt on error signals (render failures in heartbeats, device version-adoption stalling, or a spike in POS price mismatches). POST /rollouts/{id}:halt freezes advancement; rollback re-points to the previous version_id, which is instant because bundles are immutable and still cached at the edge.

Validation gates before any of that: no item priced at 0 or negative; no price change greater than X% without a second approver; every item has allergen data in every locale it's published to; every referenced asset exists; total items fit the smallest target screen. These catch the realistic disaster — a fat-fingered price rolled to 40k stores — far more reliably than a canary does, because a canary only catches what the canary stores happen to sell.

Defend it — partial fleet state. During any rollout, different stores are legitimately on different versions. That's fine for menus but not for a nationally advertised promotion, so the schedule's effective_from is what gates display, while the rollout gates distribution. Separating those two is what lets you pre-stage for days and still have every store flip together.

Fleet & distribution

• 40k stores × ~5 devices = 200k endpoints; ~5 dayparts × 2 days staged = ~10 bundles live per store.
• Manifest poll every 5 min, jittered → 200k / 300 s ≈ 670 QPS, and ~99.9% are 304s. This is a small service.
• Bundle ~5 MB (mostly images, incremental after the first). Full fleet refresh = 200k × 5 MB = 1 TB, spread over hours via CDN. Trivial.
• Resolution cost: 40k stores × 3 dayparts = 120k folds per publish, ~ms each → a few seconds on one machine.

Latency targets

• Planned change → 99% of fleet: ~15 min (3 poll intervals).
• Emergency takedown → 99%: < 30 s via websocket, with the 10 s collapsed poll as backstop.
• Daypart switch: 0 ms of network, purely local.
• Rollback: instant (re-point to a cached immutable bundle).
• Data volume overall is so small that the entire design is driven by correctness and offline behavior — say this explicitly, it shows you sized the problem.

Clarify before designing

• Boundary semantics: the examples imply [a, b) half-open — [a, aa] returns {aa} but not {aaa, ac}, and [ab, b] returns {ac}. Confirm this out loud; getting it wrong invalidates everything after.
• Result size: could a range match 10⁸ words? If yes, the API must paginate with a cursor, not return a list.
• Mutability: read-only corpus (build once, serve forever) or live insert/delete? This is the biggest fork in the design.
• Scale: 10⁶ words fits in memory on one box; 10¹¹ words does not. Ask.
• Latency SLO, and whether prefix/autocomplete queries are also needed (they change the structure choice).

Assume, and say so

• 10¹⁰ words, ~20 bytes each → ~200 GB. Doesn't fit one machine.
• Mostly reads; writes are bulk-loaded plus a live trickle.
• p99 < 50 ms for a range returning ≤ 1000 results.
• Results must be returned in sorted order and paginated.

Core tension: range queries want data sorted and contiguous; distribution wants data hashed and uniform. Those are directly opposed, and choosing sorted (range-partitioned) means accepting hot shards. Everything below follows from that.

The hot-shard problem you created

Real word distributions are wildly skewed — a shard covering [s, t) holds far more English words than one covering [x, z), and query traffic is skewed too. Range partitioning guarantees this; it's the price of ordered scans.

• Split by load, not by key space. Choose boundaries so each shard holds roughly equal bytes and QPS, not equal alphabet width. Boundaries come from sampling the actual corpus.
• Dynamic split/merge like Bigtable tablets: a shard exceeding a size or QPS threshold splits at its median key and the registry updates.
• Read replicas absorb read skew without resharding — cheap and usually sufficient, since this workload is read-dominated.

Streaming, not materializing

A range can match arbitrarily many words, so the coordinator must stream: open an iterator per overlapping shard, k-way merge them through a min-heap, emit in sorted order, stop at the page limit. Never collect the full result set in the coordinator — one range("", "") would OOM it.

cursor = base64(last_word_returned)
next page: range(cursor, b) exclusive of cursor
// stateless, resumable, survives coordinator restart

Defend it: a key-based cursor is stable under concurrent inserts, whereas an offset-based one silently skips or repeats words when the corpus changes mid-pagination.

Sizing

• 10¹⁰ words × 20 B = 200 GB raw; ~50 GB after front-coding.
• ~10 shards at 5 GB each, ×3 replicas = 30 nodes. Small.
• B+ tree fanout ~200 → depth 5 for 10¹⁰ keys; internal nodes ~1 GB, cached in RAM, so a lookup is one SSD read.
• Scan of 1000 results ≈ a few contiguous leaf pages ≈ < 5 ms.

30-second recap

Range queries need sorted, contiguous data, so I range-partition rather than hash-partition — accepting hot shards as the explicit cost. Each shard is a replicated B+ tree with internal nodes in RAM and prefix-compressed, linked leaves on SSD, so a range is one descent plus a sequential leaf walk. A coordinator consults a boundary registry, opens iterators only on overlapping shards, and k-way merges them into a streamed, key-cursor-paginated response — never materializing the full result. Skew is handled by choosing boundaries from a sample of the real corpus and splitting tablets dynamically on size or QPS, with read replicas absorbing query skew. The things I'd pin down first are the interval semantics — the examples imply half-open — and the collation, because lexicographic order is locale-dependent the moment the corpus isn't ASCII.

Functional

• search(lat, lng, radius, query?, filters?) → ranked businesses.
• Filters: category, open-now, rating, price band.
• Ranking blends distance, relevance, and quality — ask which dominates; it's a product decision.
• Business data ingest: owner edits, third-party feeds, user-suggested corrections.

Ask early: is this "find me sushi nearby" (text + geo) or "what's within 2 km" (pure geo)? The first needs an inverted index; the second only needs a spatial index. Most candidates assume one and design the wrong system.

Non-functional

• ~10⁸ businesses globally; ~10⁵ QPS peak, heavily skewed to dense metros.
• p99 < 200 ms — it's an interactive search box.
• Read-dominated ~10 000:1. Business data changes slowly; open-now and busy-ness change constantly.
• Stale results acceptable for hours on descriptions, not on permanent closure.

Core tension: geo-filtering and text-relevance want different index structures, and joining them at query time is expensive. The design is about doing the cheap filter first and the expensive scoring on a small candidate set.

Density skew is the defining problem

Manhattan has orders of magnitude more businesses per km² than rural Montana, and far more queries. A fixed S2 level is therefore wrong everywhere: at level 13 a Manhattan cell holds thousands of businesses, and a Montana cell holds none.

• Variable-level cells: subdivide until a cell holds ≤ K businesses. Dense areas get deep cells, sparse areas shallow ones. S2RegionCoverer handles mixed levels natively — that's the reason to prefer S2 over flat geohash.
• Adaptive radius: if a small radius in a dense area already returns hundreds of results, don't expand. If a rural query returns two, expand the radius progressively — but tell the user you did, or "nearest" results 40 km away look like a bug.
• Shard by cell prefix but balance by load, with hot metro cells split across more replicas. Geographic sharding without load balancing puts all of Tokyo on one machine.

Two-stage ranking, and why

L1 (cheap, on thousands of candidates):
  score = w1*exp(-dist/d0)
        + w2*log(1+review_count)*rating
        + w3*text_match_bm25
  → keep top 200

L2 (expensive, on 200):
  GBDT / neural with personalization,
  popularity-at-this-hour, click history
  → top 20

Running the expensive model on every candidate blows the 200 ms budget; running only the cheap one loses quality. The staged funnel is the standard answer and stating the candidate-set sizes is what makes it credible.

Defend it — freshness split. Static attributes (name, category, location) go in the nightly-rebuilt index. Volatile ones (open now, temporary closure, live busy-ness) must not be baked into the index — compute open-now at query time from cached hours plus the local timezone, and keep a small, fast-updating overlay for closures. Baking hours into the index means a business shows as open at 3 a.m. until the next rebuild.

Sizing

• 10⁸ businesses × ~2 KB doc = 200 GB. Index postings far smaller.
• Geo index: 10⁸ (cell_id, biz_id) pairs ≈ 1.6 GB — small enough to replicate widely.
• 10⁵ QPS × ~3 KB response = 300 MB/s out. Cache-friendly.
• Budget: coverer ~1 ms, index scan ~20 ms, L1 ~10 ms, L2 on 200 docs ~30 ms, hydrate ~10 ms → ~70 ms, well inside 200 ms.

30-second recap

The core move is turning 2-D proximity into 1-D range scans: an S2 region coverer converts (lat, lng, radius) into a handful of contiguous cell-ID ranges, which a sorted, range-partitioned index can serve directly. Cells are variable-level so a dense metro subdivides deeper than farmland, which is why S2 beats a flat geohash here. Covering is generous and then every candidate gets an exact haversine check, so boundaries can over-fetch but never miss. Text queries intersect the cell candidates with an inverted-index posting list, filters apply, then a two-stage ranker — a cheap linear pass down to 200, an expensive model down to 20 — keeps us inside 200 ms. Volatile attributes like open-now and permanent closures ride a fast overlay rather than the nightly index rebuild, because a user driving to a closed restaurant is the failure that actually matters. And I'd start on PostGIS if the scale were 10⁶ — this design only earns its complexity at 10⁸.

Functional (storage quota)

• reserve(user, bytes) before an upload; commit or release after.
• Quota is hierarchical: org → team → user, and the tightest binding limit applies.
• Deletes and trash restore adjust usage; trash counts against quota until purged (a real product decision — state it).
• Users see accurate usage, broken down by category.

Non-functional

• 10⁹ users; ~10⁵ quota ops/sec; p99 < 50 ms on the upload path.
• Accuracy over availability on the write path — the inverse of a rate limiter. Letting a user exceed quota costs real money and is hard to claw back.
• Displayed usage may lag by seconds; enforced usage may not drift permanently.

Core tension: you need a durable, monotonic, per-user counter that survives crashes and never double-counts, on a path that also needs to be fast. That's a transaction, not a cache.

The two-phase sequence

reserve(user, bytes, op_id):
  BEGIN
    row = SELECT used, reserved, limit
            FROM quota WHERE user=? FOR UPDATE
    if row.used + row.reserved + bytes > row.limit:
        ROLLBACK; return 507 Insufficient Storage
    INSERT INTO reservation(op_id, user, bytes,
              expires_at=now+1h)
         ON CONFLICT (op_id) DO NOTHING   -- idempotent
    UPDATE quota SET reserved = reserved + bytes
  COMMIT

commit(op_id):   -- after bytes are durable in blob store
  BEGIN
    r = DELETE FROM reservation WHERE op_id=? RETURNING *
    if r is null: return OK          -- already committed
    UPDATE quota SET used = used + r.bytes,
                     reserved = reserved - r.bytes
  COMMIT

sweeper (every 5 min):
  expire reservations past expires_at -> release

Why reserve-then-commit, not just increment

A single increment at upload start over-counts every abandoned upload; a single increment at upload end lets a user start a thousand concurrent uploads that each individually fit under the limit and blow past it together. The reservation makes the check and the claim atomic, which is the actual race in this problem.

op_id makes both phases idempotent, so a client retry after a timeout can't double-charge. The expiry sweeper is what stops a crashed uploader from permanently stranding quota — without it, reserved-but-never-committed bytes leak until the user is locked out of their own account with no way to recover.

Why a transactional store (Spanner/SQL) rather than Redis: this counter is the source of truth for something users pay for. It must survive a cache flush, support a real transaction across the reservation and the balance, and be auditable. This is the exact inverse of the rate-limiter argument in panel 1 — and being able to explain why the same-looking problem gets the opposite answer is the whole point of having both.

Hot row on shared quotas

An org-level quota is one row that every member's upload contends on with FOR UPDATE. A 10 000-person org serializes on it.

• Sharded counters: split the org balance into N sub-rows; a reserve picks one at random, and only when a sub-row is exhausted does it consult siblings. Reads sum all N. Cost: the "am I over?" check is approximate near the boundary, so keep a small reserve buffer.
• Lease blocks: a service instance leases 1 GB of org quota and hands out sub-allocations locally. Same pattern as the rate limiter's token lease, but here the lease must be durable and reclaimable, not best-effort.

Drift, and the audit job

Because errors are permanent, you need a periodic reconciler that recomputes true usage from the authoritative object metadata (sum of object sizes per owner) and compares it to the counter. Discrepancies get logged, and small ones auto-corrected; large ones page, because a systematic drift means a bug that's silently mischarging users.

Defend it: the reconciler is scanning billions of objects, so run it incrementally — per user, on a rolling schedule, prioritized by users near their limit and by accounts with recent anomalies. A full-fleet nightly recompute doesn't scale and isn't necessary; what matters is that every account gets reconciled on some bounded horizon.

Sizing

• 10⁹ users × ~200 B quota row = 200 GB. Sharded SQL/Spanner, easily.
• 10⁵ quota ops/s, each a short transaction on one row → ~2 000 rows/s/shard across 50 shards. Comfortable.
• Reservations in flight: 10⁵ ops/s × ~60 s mean upload = 6 M open rows. Small, and the sweeper keeps it bounded.
• Reconciler: 10⁹ users on a 7-day rolling horizon ≈ 1 650 accounts/s of background scan.

30-second recap

First I'd pin down which quota this is, because request throttling, concurrency slots, and cumulative storage need genuinely different machinery. Assuming Drive-style storage: the counter is a durable, transactional per-user balance, not a cache, because an over-count here is permanent and monetary rather than washing out at the next window. Uploads take a reserve-then-commit path — one transaction atomically checks the limit and claims the bytes, keyed by an operation ID so retries are idempotent, and an expiry sweeper releases reservations from crashed uploads so quota can't leak. Shared org quotas would serialize on one hot row, so I'd shard the balance into sub-counters or hand out durable lease blocks, accepting approximate enforcement near the boundary in exchange for concurrency. Because errors accumulate, there's a rolling reconciler that recomputes true usage from object metadata and pages on systematic drift. And the failure stance is graded rather than binary: fail open for users well below their limit, closed for users near it.

Functional

• get(key), put(key, value), delete(key); optionally cas(key, value, expected_version).
• Keys ≤ 256 B, values ≤ 1 MB.
• Optional TTL per key.
• Ask: do we need range scans? If yes the partitioning answer flips from hash to range — see panel 7.

Non-functional — pin these numerically

• 10⁶ QPS, 90/10 read/write.
• p99 < 10 ms read, < 20 ms write.
• 10 TB of data, growing 2×/year.
• Durability: no acknowledged write may be lost. Availability target 99.99%.
• Consistency: ask, don't assume. Linearizable, read-your-writes, or eventual? This single answer determines the entire replication design.

Core tension: the CAP choice isn't philosophical here, it's a product question — a session store can be eventually consistent and stay up through a partition; a counter or a lock service cannot. Make the interviewer pick, then design decisively.

From QPS to node count

Reads:  9e5 QPS.  Writes: 1e5 QPS.

Per-node capability (commodity, NVMe):
  memory hit      ~200k ops/s
  NVMe random read ~100k IOPS, ~100 us
  network          10 Gbps = 1.25 GB/s

Data: 10 TB.  Cache budget: 10% hot = 1 TB RAM.
  at 128 GB/node -> 8 nodes just to hold cache
  say 20 nodes -> 500 GB data + 50 GB RAM each

Read path:
  90% cache hit  -> 8.1e5 ops/s from RAM
  10% disk       -> 9e4 IOPS spread over 20 nodes
                 = 4.5k IOPS/node  (of ~100k) OK

Write path with RF=3, quorum W=2:
  1e5 client writes -> 3e5 physical writes
  = 15k writes/s/node, ~1 KB each = 15 MB/s
  plus WAL fsync: batch-commit groups of ~100
  -> 150 fsync/s/node, trivially fine

Network per node:
  (8.1e5/20)*1KB read + replication traffic
  ~ 40 MB/s + 45 MB/s = 85 MB/s of 1250 MB/s
  -> network is NOT the bottleneck; RAM is.

What the math is for

The conclusion — RAM for the working set, not IOPS or network, sets the node count — is the thing to say. Any candidate can propose consistent hashing; few can tell you which resource actually binds, and that's what "throughput estimation" is probing.

State the assumptions as you go and keep them round: 100k IOPS, 200k memory ops/s, 10 Gbps, 1 KB average value. Nobody is checking your arithmetic to two decimals; they're checking that you know which numbers matter and roughly how big they are.

Then use it. The math should change a decision: here it says 20 nodes is right, that we're memory-bound so a bigger cache tier is the cheapest lever, and that RF=3 quorum writes cost us nothing we can't afford. Math that doesn't change a decision is decoration.

Quorum, and the honest caveat

W + R > N guarantees the read set intersects the write set, so a read sees the latest acknowledged write. With N=3: W=2/R=2 is the balanced default; W=3/R=1 favors read-heavy workloads at the cost of write availability; W=1/R=1 is fast and eventually consistent.

Defend it: quorum is not linearizability. Concurrent writers can still produce conflicting versions, a failed write that reached one replica may later surface, and read-repair timing is not deterministic. If the interviewer wants true linearizability, say so plainly and switch to Raft/Paxos per partition with reads served by the leader (or via lease reads) — that costs you a leader election window on failover, which quorum-only designs don't have. Naming this distinction is a strong senior signal; asserting "quorum gives strong consistency" is a common and visible error.

Conflicts, hot keys, LSM cost

• Conflict resolution: last-write-wins on a timestamp is simple and silently loses data under clock skew. Vector clocks / version vectors preserve causality but push merge logic to the client. State the choice — LWW is defensible for a cache-like store, unacceptable for a shopping cart.
• Hot key: hashing spreads keys, not traffic to one key. One viral key still lands on 3 replicas. Fixes: client-side caching with short TTL, or replicating that key to extra nodes on demand. Say that consistent hashing doesn't solve this — many candidates think it does.
• LSM trade: writes are sequential and fast; reads may touch multiple levels (mitigated by bloom filters), and compaction consumes background IO and causes p99 spikes. If the workload were read-dominated with in-place updates, a B-tree would be the better pick — and knowing when not to use an LSM is the point.

The headline figures

• 10 TB × RF 3 = 30 TB physical; 20 nodes × ~2 TB NVMe. Comfortable.
• Hot set 10% = 1 TB RAM across the fleet → 50 GB/node — this is the binding constraint.
• Reads: 9e5 QPS, ~90% from memory; disk load only ~4.5k IOPS/node of ~100k available.
• Writes: 1e5 × RF 3 = 3e5/s = 15k/s/node ≈ 15 MB/s, with batched fsync at ~150/s.
• Network ~85 MB/s/node against 1.25 GB/s — an order of magnitude of headroom.

30-second recap

Consistent hashing with a few hundred virtual nodes per machine spreads both data and rebalancing cost, and replication factor 3 placed across racks and zones is where availability actually comes from. Each node runs an LSM engine — write-ahead log, memtable, SSTables with bloom filters — which suits a write-heavy path at the cost of read amplification and compaction-driven p99 spikes. Quorum with W=2, R=2 of N=3 means a read intersects the latest acknowledged write, but I'd be explicit that this is not linearizability: for that you'd want Raft per partition with leader reads, trading a failover window for stronger guarantees. Failed nodes are covered by hinted handoff and Merkle-tree anti-entropy repair. On the sizing: at a million QPS with 10 TB and a 10% hot set, it's memory that sets the node count at around twenty — disk IOPS and network both have an order of magnitude of headroom — so the cheapest lever if we need more throughput is a bigger cache tier, not more machines.